Enable dynamic filters for range-partitioned joins - #23854
Merged
alamb merged 25 commits intoAug 11, 2026
Merged
Conversation
…/hash-join-dynamic-filter-with-range-partition
saadtajwar
reviewed
Jul 24, 2026
Member
Author
|
cc @gene-bordegaray @jayshrivastava @stuhood @adriangb — kindly pinging on this one. Please take a look if you have time, thanks! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #23854 +/- ##
==========================================
- Coverage 81.01% 81.00% -0.02%
==========================================
Files 1106 1106
Lines 383974 384464 +490
Branches 383974 384464 +490
==========================================
+ Hits 311085 311433 +348
- Misses 54554 54664 +110
- Partials 18335 18367 +32 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (1a141f3) to 1763417 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (1a141f3) to 1763417 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Member
Author
@comphead We’d need to enable range partitioning on the join inputs for the effect to show up. |
getChan
reviewed
Jul 26, 2026
getChan
left a comment
getChan
left a comment
Contributor
There was a problem hiding this comment.
make sense to me!
If it's not too much trouble, could you share some performance comparison benchmark results?
Member
Author
this still need some additional plumbing to really work, but let me try to do that and run some benchmark |
Contributor
|
I believe there is much discussion going on around the future of dynamic filters here #23814 . I am unable to give PRs full attention this week but think syncing with @jayshrivastava will be very useful |
…mic-filter-with-range-partition
peterxcli
force-pushed
the
feat/hash-join-dynamic-filter-with-range-partition
branch
from
July 31, 2026 04:29
9056b30 to
bf4e6a1
Compare
|
Hi @peterxcli, thanks for the request (#23854 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: 2010YOUY01, Dandandan, Fokko, Jefffrey, Omega359, Rachelint, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, coderfender, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mkleen, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas. File an issue against this benchmark runner |
Contributor
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7d2589f) to 39d5064 (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Contributor
|
run benchmarks |
Contributor
|
run benchmark tpch tpcds |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
alamb
approved these changes
Aug 11, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Thank you for all the work @peterxcli @gene-bordegaray @getChan @jayshrivastava and @saadtajwar -- this is a pretty sweet team effort
I took brief pass through it and it looks great to me. I kicked off some benchmarks just to make sure it doesn't accidentally regress performance and assuming they look good I'll merge it in
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
Contributor
|
FYI @adriangb in case you want to review this too |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpcdsResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpchResults will be posted here when complete File an issue against this benchmark runner |
alamb
reviewed
Aug 11, 2026
|
|
||
| let filter_expr = if has_canceled_unknown { | ||
| let mut when_then_branches = empty_partition_ids | ||
| let filter_expr = if has_canceled_unknown |
Contributor
There was a problem hiding this comment.
as a follow on it might be nice to break this function into smaller functions (mostly so they can be documented more clearly) -- the techniques here are quite clever
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpchCPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark tpcdsCPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
|
Soemthing I have been thinking about. What would you guys think of creating a range partitioned or partitioned benchmarks similar to tpch or clickbench to be able to track progress on these more easily? I dont know how serious of an endeavor this would be and we can also lookat what is already available but I think it would be highly valuable |
Contributor
Yes I think that would be a great idea. One thing you could perhaps do is use the TPCH queries but repartition the inputs by time (similar how it is done for tpch_sort) |
Contributor
|
run benchmark clickbench_partitioned |
Contributor
@alamb ok great. I was planning on creating a new wave of work for range partitioning like optimizations and benchmarks now that the base is in and this will definitely be on the list. I will create issues for these things and can start some discussions there 👍 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark clickbench_partitionedResults will be posted here when complete File an issue against this benchmark runner |
Member
Author
@alamb @gene-bordegaray I created a rough version earlier, happy to refine it together later if that's useful. |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/hash-join-dynamic-filter-with-range-partition (7274de3) to a05388e (merge-base) diff Run configurationrun benchmark clickbench_partitionedCPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
@peterxcli this is great I will check it out and definitely be part of next wave of work for range |
Contributor
|
Merging this one in to get us ready for the 55 RC |
peterxcli
deleted the
feat/hash-join-dynamic-filter-with-range-partition
branch
August 12, 2026 07:03
kosiew
pushed a commit
to kosiew/datafusion
that referenced
this pull request
Aug 12, 2026
## Which issue does this PR close? - Closes apache#23376. ## Rationale for this change Partitioned hash joins build one dynamic filter per build partition. Existing routing uses `hash(key) % N`, which cannot reproduce a Range partitioning layout. Compatible Range co-partitioned joins instead need to route probe rows using their existing ordering and split points. ## What changes are included in this PR? - Enable dynamic filter pushdown for hash joins with compatible Range-partitioned inputs. - Build a searched `CASE` expression that routes probe rows to the corresponding partition filter using the Range ordering and split points. - Move the TopK lexicographic filter builder into the shared ordering module for reuse. ## Are these changes tested? unit test. ## Are there any user-facing changes? Yes. This PR adds `RangeExpr` to the physical-expression protobuf model, which adds the public `ExprType::RangeExpr` enum variant. Downstream Rust consumers that exhaustively match `ExprType` must handle the new variant. It also enables dynamic-filter pushdown for compatible Range-partitioned joins.
Member
Author
Thanks @gene-bordegaray. I also commented on a related “signed zero” edge case in: #23854 (comment). Might be good to include in the next wave of work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Partitioned hash joins build one dynamic filter per build partition. Existing routing uses
hash(key) % N, which cannot reproduce a Range partitioning layout.Compatible Range co-partitioned joins instead need to route probe rows using their existing ordering and split points.
What changes are included in this PR?
CASEexpression that routes probe rows to the corresponding partition filter using the Range ordering and split points.Are these changes tested?
unit test.
Are there any user-facing changes?
Yes. This PR adds
RangeExprto the physical-expression protobuf model, which adds the publicExprType::RangeExprenum variant. Downstream Rust consumers that exhaustively matchExprTypemust handle the new variant.It also enables dynamic-filter pushdown for compatible Range-partitioned joins.